typo Kuberentes /Kubernetes

s/focus/focuses/

I'd recommend focusing on one feature at a time. If that's gang scheduling, just talk about how k8s would enable it either via default scheduler or a new scheduler under incubation.

s/maybe/may be/
s/kind/type/ ?

I think you should remove this field from the template. Priority is supposed to be resolved at the time of Pod creation. It exists in PodSpec, but it doesn't make sense to be in the template. This field is not expected to be copied to PodSpec.

Reusing Priority for "Job" here; and the priority of Pod will only take effect within a "Job".

There's the requirements on Job priority, IMO: kubernetes/kubernetes#65746

NodeSelector is in a way an earlier and inferior version of NodeAffinity. Why not use NodeAffinity instead of NodeSelector?

Agree, NodeAffinity is better; using NodeSelector because of simplification in prototype :)

s/defines necessary parameters of Job for scheduler/specifies the scheduling constraints of a Job/
s/minimal/minimum/

Given that this type shows status of a job, shouldn't this field be NumAvailable?

s/consider/considered/

Do we plans to support running long-running services as a gang (QueueJob) as well? If we don't, we may not need to support rolling updates for batch jobs which are expected to run to completion.

Do we plans to support running long-running services as a gang (QueueJob) as well?

Good question! I used to plan to support long-running services too; but for now, did not get requirement right now. So I'd like to make it batch work only firstly to make first release simple.

/cc @mindprince , fyi, some changes since we last sync up.

How do you deal with the last set of running jobs? For example, let's say that minAvailable is 20. Total number of tasks in the QueueJob is 100. 85 of them have already run and finished. 15 of them are still running. 15 is less than minAvailable, but we should let them run.

Yes; it need to include Succeeded pods: Succeeded + Runinning <= minAvailable

You haven't described users' workflow here, but I guess users only need to create a QueueJob and then the kar-controllers create Pods and SchedulingSpec. Is that right?
If so, why do we need SchedulingSpec? QueueJob has SchedulingSpecTemplate already and kar-scheduler can get MinAvailable from there. Other fields of SchedulingSpecTemplate will be available in the PodSpec of each Pod created for a QueueJob.

You haven't described users' workflow here, but I guess users only need to create a QueueJob and then the kar-controllers create Pods and SchedulingSpec. Is that right?

That's right !

If so, why do we need SchedulingSpec?

SchedulingSpec is used for customized controller, e.g. kubeflow/tf-operator. I do suggest to use QueueJob, but I do not want to make it as restriction currently. For example, kubeflow/tf-operator can using this by replacing PDB with SchedulingSpec.

We also need to cover admission in this document. I think we ideally want to admit either the whole gang or none of the members of a gang. Doing so looks challenging to me with the existing admission controllers. Do you have any ideas on addressing this issue?

Transactional multiobject admission is not possible.

Would this mean that we have to necessarily have a wrapper (maybe QueueJob) that will allow for admission of a gang of pods in one shot?

by the time scheduling happens, the pod objects have already been admitted, which means object counts, resource quotas, and pod spec restrictions have already been enforced... if the gang scheduler is just matching up pods to nodes, it doesn't seem like admission would cause a lot of problems

Would this mean that we have to necessarily have a wrapper (maybe QueueJob) that will allow for admission of a gang of pods in one shot?

no, you can't end-run around individual pod admission via another object

I think we ideally want to admit either the whole gang or none of the members of a gang.

It's better to delegate this to controllers or Applications; it's hard for admit to collect "dynamic info", e.g. for ResourceQuota, it's hard for admit to decide which pod should be create to meet 'Gang/PodSchedulingGroup' as much as possible. In google doc, the QueueJob will stop creating new Pod if rejected (there're several corner case to cover)

In google doc, the QueueJob will stop creating new Pod if rejected (there're several corner case to cover)

The problem is that the user experience would be poor. A few pods of a gang are created, but the rest fail admission and are not created. Those pod created will not be scheduled, but will consume the user's quota. I guess the user has to delete them manually. If we cannot have transactional admission, at lease we should have a controller clean up such pods.

A few pods of a gang are created, but the rest fail admission and are not created.

For batch or run-to-complete workload, it's better to have some pending jobs there; so k8s can start the pending jobs immediately when some running jobs finished, instead of waiting for client to submit new jobs; hmm...., a kind of pipeline. The difference is that
the "controller" has to make sure not introduce deadlock (e.g. for every pod group, just start part of them), only the "last" pod group should be impacted.

For example, in google doc, the QueueJobController will create pods one by one based on QueueJob's minAvaialbe; if minAvaialbe pods are not created, it'll NOT create pods for next QueueJob. If all minAvailable are meet, QueueJobController will create pods up to .spec.replicas in round-robin for each QueueJob. There're still some cases are not covered, e.g. update minAvailabe for a better SLA, new high priority job, start other jobs if can not meet minAvailable. That's another reason that prefer to focus on batch/run-to-complete firstly, and introduce QueueJob; I'm also ok to enhance Job to support those cases in future, and maybe Queue.

Currently, AdmissionController checks pod based on the 'snapshot' of status/config, e.g. ResourceQuota. But for gang scheduling, it also need to check whether controller WILL create more pods later to meet gang requirement. And a wrapper object can not help as some AdmissionController are still doing pod level check.

Another corner case is that: a hug job (e.g. 1000+ tasks) waits for long running run-to-complete jobs (e.g. hours or days) :(
The candidate solution is to support Queue, and balance the resource sharing between Queue. This's also in backlog.

I have a slightly different idea about the API. Can we create a namespaced object, let's call it a gang, that only specifies properties of a gang. Something like:

type Gang struct {
    metav1.TypeMeta
    metav1.ObjectMeta

    NumMembers int
    Policy GangLifeCyclePolicy
}

Then Pod.PodSpec can have a field called GangName or GangId. When the field is not empty, it means that the pod is gang member. When scheduling pods which are gang members, scheduler makes sure that at least Gang.NumMember of such pods can be scheduled simultaneously or none of them are scheduled.

The Gang type can also include gang life cycle policies. For example, the policy can specify that when the number of running pods drops below NumMember kill all the other instances, or wait for a certain amount of time for the gang members to reach NumMember before killing other instances, ....

Here are some of the benefits of this design:

1. Existing workloads that use one of the K8s collections, e.g. ReplicaSet, Job, etc., can keep their config and only add the GangName to their pod template.
2. Flexibility. Pods of a gang can have different PodSpec and belong to different collections.
3. Existing controllers which are responsible for managing life cycle of collections work well with this design.
4. Users who have already learned and worked with K8s collections can keep using the familiar collections.

This sounds like something that should be a label before we create a field on pod spec (also gang wouldn’t be a great name for an api object).

We need a way to identify a gang where labels could suffice in theory. We also need to know how many pods comprise a gang (or minimum size of a gang). Encoding that value isn't ideal.
I'd recommend modifying @bsalamat's proposal slightly by including a label selector to identify pods belonging to a gang.

+1 for aiming to have compatibility with existing controllers, Job API at minimum.

Used to propose PodSchedulingGroup in the google doc, so I totally agree with you to provide such kind of API object in core.

Regarding label selector and 'GangID/GroupID', there're some cases we need to handle:

• race condition: when a scheduler got a Pod w/o 'Gang/PodSchedulingGroup', how to schedule it?
  • If schedule the pod and no 'Gang/PodSchedulingGroup' coming, that's OK
  • If schedule the pod, 'Gang/PodSchedulingGroup' coming later and meet 'minAvailable', that's OK
  • If schedule the pod, 'Gang/PodSchedulingGroup' coming later but less than 'minAvailable':
    • if it's because of controller delay, schedule should wait
    • if there's no pod anymore, schedule should handle it by LifeCyclePolicy
  • If does not schedule the pod, how long should we wait?
• performance: for a batch or run-to-complete workload, the pod come and go in short time; it may take time to
  build relationship between 'Gang/PodSchedulingGroup' and Pod
• summary/status: if 'Gang/PodSchedulingGroup' support cross Kind/Types, it's better to provide a way
  show the summary/status, such succeeded, running, pending

Here's what I did in kube-arbitrator:

• race condition: as a batch scheduler, kube-arbitrator do scheduling based on 'job'; so it'll not schedule pods
  until get 'Gang/PodSchedulingGroup'
• performance: using OwnerReference (controller==true); so 'GangID/GroupID' will be better for performance. 'label selector' is
  kind of short term solution to avoid touching more Kind/Type
• summary/status: it delegate 'Gang/PodSchedulingGroup' owner to handle summary/status

Another minor comment is the name: as you known, Gang-scheduling is one of requirements, we may enhance this object for other requirements later; so a more
general name maybe better, also including GangLifeCyclePolicy, e.g. changing to PodGroupLifeCyclePolicy. And suppose LifeCyclePolicy is going to be handled by kube-arbitrator.

I totally agree with @k82cn on label selector. I think a pod should identify itself as a gang member, not the label selector of a gang choosing pods. Klaus provided the reasons, so I don't repeat them again.
(Feel free to change the name of the object. I agree that Gang is not a good name for our API. I just used it to present the idea.)

+1 to general point raised by @bsalamat here!

For naming, how about something simple like SchedulingGroup? Then the gang-ness can be specified in the policy.

If you were to consider the approach @bsalamat suggested, we would not need a queue job I think.

Yes; if so, I'd like to enhance 'Job' instead of introducing a new Kind.

For example, multiple pod template (or wrapper of multiple job).

s/MinAvailable/NumMember/g

NumPods?

Minimum? It's likely rare to set less than Replicas, so this should be optional for simplicity.

@beberg it may be needed in many cases for a Job or ReplicaSet, etc. to have more replicas than those needed to be run together.

It's likely rare to set less than Replicas, so this should be optional for simplicity.

PodGroup is a common concept for "batch" feature; other Kind/type can also use that.

s/Template/Spec

Done :)

RetryCount is universal enough it should be specified now to prevent a proliferation of solutions. HPC workloads are prone to failures and checkpoint/restart mechanics are common. RestartPolicy Never leads to far too many failed jobs that could have been resumed automatically (overnight, weekends, ...), so PodSchedulingGroupTemplate.RetryCount is needed.

We'll discuss this later; IMO, RetryCount is a common requirement, not only to batch.

Since the context and point of this API is gang scheduling, "Restart" is the more logical default here for both "PodFailed" and "Unschedulable". The none/none can be done with exiting APIs today. Would also like to know the use case in mind for restart/none and none/restart, as they seem to be non-gang scenarios covered by existing APIs.

Would also like to know the use case in mind for restart/none and none/restart

What does "restart/none and none/restart" mean?

What do you think about renaming this to just PodGroup?

+1

Done

@beberg it may be needed in many cases for a Job or ReplicaSet, etc. to have more replicas than those needed to be run together.

I think we should also add a Timeout field here. When Timeout is specified, the controller waits for the Group members to satisfy the Group requirements. If the timeout expires, it takes the Action.

What does PodFailed mean in this context? Should this field instead be a Phase that indicates the state of the gang/group?

we should also add a Timeout field here.

+1, and Done

What does PodFailed mean in this context? Should this field instead be a Phase that indicates the state of the gang/group?

That means at least one Pod of Group are failed; PodGroupController need to take action accordingly. Did not introduce Phase for PodGroup as only the Pod's "controller" knows phase; for example, for a batch job, there maybe Failed, Done, Suspend, Terminated and so on Phase; PodGroupController does not know that :(.

nit: s/support/supported/

please replace numMembers with minMembers here and below.

It is not clear what this item means. What do you mean by "reconstructing job"?

s/moing/moving/

Today, we do not delete completed pods as long as their parent object (such as Job, ReplicaSet, etc) is not deleted. This behavior is essential for the proposed algorithm to work correctly.

(This is a note to ourselves. Nothing needs to be changed in the design.)

Is the minMembers field in PodGroup expected to track the number of non-terminated pods or all pods that have been successfully admitted?

Vish, I think "succeeded" means those pods that have run and finished successfully. This makes sense for "run-to-completion" pods.

I would change the last sentence to
"Users and cluster admins may reduce the probability of such conflicts by partitioning the clusters logically, for example, by placing node-affinity to distinct set of nodes on various groups of pods."

nit: Should it be incubator.scheduling.k8s.io? That way once it's out of incubation we only need to drop a sub-domain?

Yours seems better :)

Why is tracking Succeeded and Failed useful at the pod Group level? This seems to make an assumption that pod groups are typically used only for run to completion jobs.

Just give a summary of pod status here; not only for completion jobs.

nit: Mark it alpha explicitly & be consistent with the object kind - alpha.incubator.scheduling.k8s.io

hm .... , mentioned this alpha feature in doc and release notes (in future); so when we migrate this feature to upstream, the user did not need update annotation in pod.